On the Sample Complexity of Predictive Sparse Coding

Predictive sparse coding algorithms recently have demonstrated impressive performance on a variety of supervised tasks, but they lack a learning theoretic analysis. We establish the first generalization bounds for predictive sparse coding. In the overcomplete dictionary learning setting, where the dictionary size k exceeds the dimensionality d of the data, we present an estimation error bound that is roughly O( √ dk/m + √ s/(μm)). In the infinite-dimensional setting, we show a dimension-free bound that is roughly O(k √ s/(μ √ m)). The quantity μ is a measure of the incoherence of the dictionary and s is the sparsity level. Both bounds are data-dependent, explicitly taking into account certain incoherence properties of the learned dictionary and the sparsity level of the codes learned on actual data.